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APPLICATION OF HUMAN FACIAL FEATURES RECOGNITION TO 

AUTOMOBILE 
SECURITY AND CONVENIENCE 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates generally to an imaging system in a vehicle for 
detecting the presence of a person and, more particularly, to an imaging system 
5 for use in a vehicle that identifies a human face, and uses this information for 
vehicle security and convenience purposes. 

2. Discussion of the Related Art 

The National Highway Traffic Safety Administration requires passenger 
side airbags on all .passenger cars manufactured after 1998. Airbags have saved 

10 thousands of lives, but safer airbag deployment systems can save even more 
lives. Proposals have been submitted by the National Highway Traffic Safety 
Administration to require advanced airbags that are safer and more efficient. 
Thus, there is a need in the art for an advanced airbag deployment system that 
determines airbag deployment depending on the detection, identify and tracking of 

15 a person in the passenger seat of the vehicle. 

Future airbag deployment systems will be able to identify and track a 
person in the passenger seat of the vehicle. Based on this information, the 
system will provide signals whether to fire the airbag, fire the airbag at low velocity 
or fire the airbag at normal high velocity during a crash event, depending on 

20 whether a person is seated in the passenger seat, the size of the person and the 
position of the person relative to the airbag deployment door. In one example, if a 
person is detected in the passenger seat, and is some small distance (for 
example, within three inches) from the airbag door, then the airbag does not fire 
during a crash event. If a person is detected in the passenger seat, and is close 

25 (for example, between three and eight inches) to the airbag door, the airbag is 
fired at a lower velocity during a crash even. If the person is detected in the 
passenger seat, and far enough away (for example, more than eight inches) from 
the airbag door, then the airbag is fired at normal high velocity during a crash 
event. 

1 
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Present passenger sensing systems typically include ultrasonic, weight, 
infrared and/or electromagnetic sensors to detect a passenger for airbag 
deployment. Ultrasonic sensing systems sense the motion of objects within the 
passenger seat area and determine whether the object is moving closer or farther 
5 away from the sensor location. However, ultrasonic sensors don't identify the 
nature of the object, and thus can't tell if it is a person or some other object, such 
as a bag of groceries. Similar to ultrasonic sensors, microwave sensors employed 
in active doppler radar systems can track objects, but cannot identify human 
presence. Weight sensors identify objects in the passenger seat based on applied 

10 pressure, but don't consider-the passenger's location relative to the airbag. 
Passive IR sensors acquire thermal images of the passenger seat, but these 
systems are very costly. Present active IR sensors sense the relative location of 
the passenger relative to the airbag, but cannot identify human presence. 
Electromagnetic systems include LC resonant circuits where body capacitance is 

1 5 used to detect presence and identify objects, but these systems can not track the 
objects. 

U.S. Patent No. 5,835,613 issued to Breed et al., November 10, 1998, 
discloses a vehicle interior monitoring system that claims to identify, locate and 
monitor persons in the passenger compartment of the vehicle. The monitoring 

20 system employs infrared emitters that illuminate the interior of the vehicle, and 
charge couple device (CCD) arrays that detect the radiation. Outputs from the 
CCD arrays are analyzed by computational devices that employ pattern 
recognition algorithms to classify, identify or locate the content or objects in the, 
passenger seat. The pattern recognition system for determining vehicle 

25 occupants disclosed in the '613 patent employs complicated software that must 
learn the shape of an individual in all kinds of lighting situations under various 
conditions. Additionally, employing pattern recognition in this manner is limited in 
its ability to track the individual as he or she moves around in the passenger seat. 
Further, the ability to identify and track humans by general pattern recognition is 

30 questionably unreliable. Pattern recognition cannot identify who the person is, 
only detect an object's shape. 
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What is needed is an improved airbag deployment sensor and system that 
is able to effectively identify and track a person in the passenger seat of a vehicle. 
It is therefore an object of the present invention to provide such a sensing system 
for vehicle passenger airbag deployment 

SUMMARY OF THE INVENTION 
5 In accordance with the teachings of the present invention, an imaging 

system for providing vehicle security and convenience features is disclosed that 
employs face recognition software and algorithms to identify and track a person. 
The system includes infrared emitters that emit an infrared signal along a 
predetermined field-of-view, and an infrared sensor, such as a CMOS sensor used 
10 as a video signal array, that receives reflected infrared illumination from objects in 
the field-of-view. A processor, including the face recognition software, is 
employed to detect human faces. Once a face is detected, it can be compared to 
a data base to identify the person. 

Various applications of the imaging system for providing driver convenience 
15 and security include determining driver identification as the driver approaches the 
vehicle for automatically unlocking the door and starting the vehicle; determining if 
a potential thief is in the vehicle by face recognition; and providing driver seat 
adjustment, rear and side mirror adjustment and steering wheel adjustment. 

Additional objects, advantages and features of the present invention will 
20 become apparent from the following description and appended claims, taken in 
conjunction with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 is a side, cut-away, plan view of a person in the passenger seat of 
a vehicle in connection with an image sensing system for airbag deployment, 
25 according to an embodiment of the present invention; 

Figure 2 is a perspective view of a video camera and LED illuminator unit 
employed in the airbag deployment system of the present invention; 

Figure 3 is a graph with wavelength on the horizontal axis and luminous 
energy on the vertical axis showing the luminance curve for sunlight; 
30 Figure 4 is a representation of camera orientation and position parameters 

with respect to the center and normal of the airbag door, 
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Figure 5 is a block diagram of the airfaag deployment system of the present 
invention; 

Figure 6 is a schematic diagram depicting a frame differencing technique 
used in the airbag deployment system of the present invention; 
5 Figure 7 is a timing diagram for the frame differencing technique of the 

invention; 

Figure 8 is a top plan view of a vehicle showing various sensor locations for 
sensing different areas of the vehicle for various security and convenience 
purposes, according to an embodiment of the present invention; 
10 Figure 9 is a side view of a vehicle showing a camera having a field-of-view 

for detecting objects in front of the vehicle, according to another embodiment of 
the present invention; 

Figure 10 is a top view of a vehicle including a side camera and a rear 
center camera for detecting objects at the side and rear of the vehicle, according 
15 to an embodiment of the present invention; 

Figure 1 1 is a side view of a vehicle including a camera positioned in front 
of the driver of the vehicle, according to another embodiment of the present 
invention; and 

Figure 12 is a side view of a vehicle including a camera mounted in the 
20 dome light of the vehicle for intrusion purposes, according to another 
embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The following discussion of the preferred embodiments directed to an 
airbag deployment system that employs human facial feature recognition is merely 
25 exemplary in nature, and is in no way intended to limit the invention or its 
applications or uses. 

According to one embodiment of the present invention, a technique for 
detecting, identifying and tracking a person in the passenger seat of a vehicle is 
disclosed. As discussed above, it is sometimes desirable to deploy a passenger 
30 side airbag in a vehicle at low velocity, or not deploy the airbag, during a crash 
event, depending on whether a person is occupying the passenger seat of the 
vehicle, the proximity of the person to the airbag door, and the size of the person. 
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According to the present invention, the identification and tracking of the person is 
determined by face recognition software, in particular, software that recognizes 
and tracks the person's eyes and other facial features. Tracking can take place 
that way at many head angles and poses. The software algorithm would use 
5 calibrated face and feature separation to estimate range. The tracking of a 
localized area on the human face allows more image frames to be acquired per 
second, permitting more frequent tracking of the person. The more frequent 
tracking of the facial features will prevent acquiring a blurry image because the 
image is updated more frequently. 

1 0 Various software is known in the art that processes data from video data 

patterns received from an object being analyzed, and determines whether the 
object has a face. For example, such software includes the Visionics Face-It 
software, well known to those skilled in the art. it is stressed that the present 
invention is not limited to any particular facial feature mapping function, but can 

15 include any known algorithm, suitable for the purposes described herein, for 
recognizing facial features, whether it be two-dimensional or three-dimensional, 
that are then also to be used for ranging functions' as well. Further, according to 
the present invention, ranging algorithms are used in combination with the know 
face recognition software. 

20 As will be discussed in detail below, the present invention employs infrared 

radiation reflected off of objects in the passenger side of the passenger 
compartment of a vehicle that is received by an electronic video camera. The 
video camera generates the electrical signals and image used by the face 
recognition software to determine the presence identify and tracking of the person. 

25 Figure 1 is a cut-away, side, plan view of the passenger side compartment 

10 of a vehicle 12. In this depiction, a person 16 is shown in the passenger seat 
18, where the person 16 is moving forward during a crash event. An airbag pillow 
20 is shown being deployed through an airbag door 24 housed in an instrument 
panel 22 during the crash event. 

30 A video camera and IR LED illuminator unit 26 is mounted on the 

instrument panel 22 at a location suitable for the purposes described herein. 
Figure 2 is a perspective view of the unit 26 removed from the vehicle 12. The 
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unit 26 includes a cluster 28 of I R LEDs 30. A plurality of LEDs 30 is provided to 
generate the necessary intensity for daylight operation. The cluster 28 emits a 
beam of IR radiation towards the person 16 that is reflected therefrom back 
towards the unit 26. A video camera 34 is provided in the unit 26 to receive the 
5 re f)ected radiation from the person 16. The video camera 34 is used by way of a 
non-limiting example in that any detector that detects infrared radiation suitable for 
the purposes described herein can be used. 

A filter 36 is provided over the camera 34 to filter out radiation that is not 
within the desirable infrared range. The filter 36 can be any filter suitable for the 

10 purposes described herein, such as a Ti0 2 filter or a polarizing filter. The filter 
layers and thickness can be chosen to transmit the IR image to the detector, but to 
reflect the visible image away from the detector. The polarizing filter can be used 
to reduce visible light to the detector using electro-optical polarization that passes 
the IR wavelengths, but strongly attenuates the non-IR wavelengths. Figure 3 

15 shows the luminance curve for sunlight, where the filter 36 passes infrared 
radiation in the 40 nm bandpass window. The filter 36 provides some protection 
against sunlight that may affect the operation of the airbag deployment system, 
and recognition of the face of the person 16. 

In this embodiment of the present invention, a single camera is used to 

20 acquire and monitor the range of the person 16. The software employed to 
perform this function utilizes two separate locations on the occupant's face to 
provide the ranging. In a preferred embodiment, it is the person's eyes that are 
detected to provide the triangulation for ranging purposes. However, as will be 
appreciated by those skilled in the art, other facial features of the person 16 can 

25 also be used, such as the person's ears, etc. Also, the software algorithm allows 
the person's head size to be determined so that both eyes do not need to be in 
view to track the person after he has been acquired. Additionally, the software 
can be used to view other parts of the person's bodyi such as the person's torso, 
in combination with the facial feature or head size detection. 

30 Because human facial features are unique, a database can be used to 

store specific information, such as eye-to-eye separation, about a person, so that 
the software can particularly identify that person. This is important, in one 



XXID: <WO _ 0208022A2 .!_> 



WO 02/08022 




CT/US01/21687 



example, so that the system can identify children and fifth-percent females, and 
inhibit air bag firing for these people, as required by government mandated 
regulations. Also, being able to specifically identify a person improves the ranging 
accuracy of the system because the system knows that person's eye separation 
5 or other specific facial features. 

The acquisition and tracking software needs to be calibrated for a particular 
seat position, airbag position, vehicle makeup, etc. Figure 4 is a representation of 
the camera orientation and position parameters with respect to the center and 
normal of the airbag door 24. The three parameters that must be estimated in the 

10 vehicle for ranging according to the invention after the camera and Visionics Face- 
It software are calibrated include two position offsets and one angular offset. The 
two position offsets are the lateral offset of the camera 34 with respect to the 
center of the airbag (AY), measured laterally and perpendicularly to the normal 
vector out of the center of the airbag door 24, and the front or back offset (aX) of 

15 the camera 34 with respect to the center of the airbag door 24, measured along 
an axis parallel to the normal out of the center of the airbag door 24. The angular 
parameter is the azimuth offset (AG) between the optical axis of the camera 34 and 
the normal vector coming out of the center of the airbag 20. Only two of these 
parameters, aX and a<D, are used in the modified monocular distance equation. 

20 A calibration procedure, described below, can be used to determine aY, aX 

and AO. There is a trade off between measuring a large number of calibration 
points and linearly interpolating between them or measuring a small number of 
calibration points and non-linearly interpolating between them. In theory, 
calibrating with a large number of points would not require a model for estimating 

25 that is described herein. The cost of making the large number of measurements 
necessary for a brute force calibration and the possibility that out of range 
operation of the camera 34 or software might occur unnoticed with brute force 
calibration must be considered before using that approach. The approach used 
herein uses a small number of calibration tests in order to form a model for 

30 interpolation. ^ 

From Figure 4, the following equations can be written *in order to relate 
measurements made in the airbag door reference frame (xyz) to those made in 



XXID: <WO 0208022A2J_> 



WO 02/08022 ^^-CT/USOl/21687 



the camera lens reference frame (xyz). Only face positions in front of the airbag 
door (x>o) are considered in the following equations. For p ,e (pitch or elevation 
angle) and <j> (yaw or azimuth angle) defined in the camera's spherical coordinate 
system, the equations for displacements x and y in the airbag coordinate system 
are: 

x=-AX+psin(e+Ae)cos((|>+A<i>) ( 1 ) 

y=-AY+psin(e+A8)sin((j)+A<j)) (2) 
This assumes that 0 + A9 is fixed during the calibration. Rearranging equations 

(1)and (2) yields: 

Z±**L = tan(<* + A<D) (3) 
x + AX 

Making eye coordinate readout measurements at fixed x+aX. and then taking the 
slope of the tangent with respect to changes in y yields: 



10 



1 



fixed _x= atm * + * V) . (4) 

dy 



d tan(^ + AO) 

x + AX 

Knowing x and the result on the right side of equation (4), aX can be determined. 
15 Knowing aX, equation (3) and the measured data can be used to determine aY. 
Then, using aX and aY, equation (3) and the data, a$ can be determined. Using 
equation (3), and the fact that: 

eye_separation_distance=y rt _eye-yteft_eye. ( 5 ) 
the modified monocular equation follows from equation (6) for defining the 
20 objective parameter x, or the eye to airbag distance. 

eye _ separation _ dis tan ce 



x = -AX + 



(6) 



tan^ lefleye + A<t>) - tan(<zJ r , + AO) 

The calibration table consists of a lookup table of the SDK determined eye 

coordinate as a pixel value linked to the associated ray slope, as viewed in the 

camera coordinate system. Equation (6) can be simplified in terms of ray slopes 

25 (tangents of singular angles), so that the calibration table can be used directly to 

determine the eye to airbag distance from monocular operation. Employing the 

tangent trigonometry identity in equation (7) below, 

tan(a) + tan(6) , 7 s 

tan(a + b) = — (7 ) 

l-tan(a)tan(6) 

8 
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15 



20 



25 



and applying this identify to equation (6), leaves the modified monocular equation 
in a form that can be used by direct look up of tangent/slopes from the calibration 
lookup table. This equation is given in equation (8) below. 



Figure 5 is a block diagram of an imaging system 50 of the invention 
including a digital signal processor (DSP) 52. The DSP 52 includes the face 
recognition software and ranging functions that performs the analysis on the 
image generated by the camera 34. A clock generator 60 provides timing for the 
various digital devices in the system 50, and a power management system 62 
provides the power. The DSP 52 is connected to a CMOS chip 70 including a 
pixel array 72 that represents the IR detector, such as the camera 34. In this 
example, the pixel array 72 includes 256x256 pixels to provide the desired level of 
resolution. The CMOS chip 70 also includes various elements for timing and 
control purposes, including state matching logic 74, clock/timing circuitry 76, 
analog conditioning circuit 78, registers/buffers 80, on-chip programmable logic 
circuitry 82, etc. Additionally, an analog-to-digital converter 84 is also provided to 
convert the analog signal from the pixel array 72 to a representative digital signal. 
An on-chip SRAM memory 86 is shown for storage purposes, but can be off-chip 
as well. The operation of these devices in the system described herein would be 
apparent to those skilled in the art. 

In one embodiment, the infrared LEDs 30 are continuously on to provide 
the reflected radiation received by the camera 34. However, typically some kind of 
filtering or signal processing must be done to correct for the problems caused by 
direct sunlight on the camera 34 that make it through the filter 36. Particularly, the 
system 50 needs to be able to distinguish between shadows caused by sunlight 
and actual edges of features on the occupant 16. According to the present 
invention, a frame differencing technique is employed that synchronously pulses 
the LEDs 30 on for a predetermined period of time and a predetermined number 
of frames of video data, and then off for a predetermined period of time over the 
same number of frames of video data. Then, the frames of data are subtracted 



eye _ separation __ dis tan ce 



x = -AX + tan(<zU ^) + tan(AO) tan(^ r ,_^) + tan(AO) 



(8) 



l-tan(^ /fi// Man(A<D) l-tan(^ ^)tan(AO) 
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from each other so that the frames without IR illumination can be subtracted from 
the frames with IR illumination, and the background can be eliminated. The 
detector is electronically shuttered synchronously with the pulses to provide 
exposure control. The frame differencing technique described herein is used in 
5 conjunction with the infrared pulsing to achieve the desired result. In other words, 
the frame differencing is synchronized to the infrared pulses. 

The concept of frame differencing is the time-aperture, pixel-level storage 
of images using natural illumination only and natural illumination with added 
infrared illumination. Frame differencing allows these images to be subtracted to 

10 mitigate the effects of strong visible illumination. The set-up includes a neutral 
density filter that sets the IR illumination, plus worst case background to maximize 
analog-to-digital converter input. Face recognition requires that the worst case 
analog-to-digital range for the differenced image be 5 to 6 bits. The visible light 
would fit within the remaining range allowed by the analog-to-digital converter. 

15 The image differencing is either performed in the analog domain, where two pixel 
level capacitors are charged, one at each illumination level, or in the digital domain 
where a RAM memory of the digitized pixel output is taken at each illumination. 
The frame differencing acts to subtract out the illumination effects of the visible 
illumination, and to improve the image contrast. The frame differencing function 

20 can be performed in the acquisition/high bandwidth generation mode or in the 
narrow bandwidth track mode using pulsed LED illumination. The number of the 
electrons from the pulsed IR light source must be 10 times greater than the photon 
noise of the ambient illumination. Here, the noise of the ambient illumination is the 
square root of two times the number of electrons within the sun intensity, because 

25 two image frames are being acquired for every one IR image received. 

Figure 6 is a representation of how the frame differencing is performed in 
the camera 34, according to one embodiment of the present invention. Figure 7 is 
a signal timing line showing the operation of the frame differencing technique of 
the invention. A pixel array 90 in the camera 34 receives radiation from the scene 

30 for a predetermined time period (10 jas) during a pulse of IR from the LEDs 30. At 
this time, the pixel array 90 receives ambient light and infrared light. The charge 
stored by each pixel or photodiode 92 in the array 90 is then transferred to a 

10 
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charge storage site 94 made up of a plurality of capacitors, one capacitor 96 for 
each pixel 92. About 10 j^s later, at the end of the pulse from the cluster 28, the 
pixel array 90 detects just the ambient light for the same time period. The charge 
received by the pixel array 90 at this time period is stored in a capacitor storage 
5 site 98, having capacitors 100. An electronic shutter is employed in the detector 
to open and close at the appropriate time synchronously with the pulses of IR 
radiation for the operation described herein. 

The two storage sites 94 and 98 are summed in a summation amplifier 102. 
The difference between the two storage sites 94 and 98 is then digitized by an 
10 analog-to-digital converter 104, and represents the frame of data where the 
ambient light has been removed. The readout of data takes about 10 ms, then at 
the next time period, the next pulse from the cluster 28 occurs. The complete 
frame differencing process can be performed on a single chip in CMOS where the 
pixel array 90 and the storage sites 94 and 98 are together. In an alternate 
15 embodiment, the frame differencing is performed at a different time period at an 
off-chip site, where the storage sites 94 and 98 are RAM. 

The frame differencing technique of the invention can be described in the 
following manner. The variables are defined as ambient illumination l(x,y), direct 
ambient illumination T(x,y), scene reflectance R(x, y), and modulated source as L 
20 when on, and 0 when off. The response of the camera 34 is proportional to the 
product of reflectance and illumination. 

S(x,y,OFF) = k*(l(x,y)*R(x I y))+T(x,y) 
S*(x,y,ON) = k*((L+l(x,y)rR(x,y))+T(x,y) 
D(x,y) = S(x,y,ON) - S(x,y,OFF) = KL*R(x,y) 
25 This difference scene has much smaller dynamic range than the simple image 
S(x,y,OFF). The same benefit can be derived by reading out the frame with LED, 
then 

reading the frame without LED and subtracting the frames external to the camera 
34. The penalty is increased dynamic range required to avoid saturation. 
30 L must be much greater than the direct ambient illumination response I. 

Thus, I is made as small as possible by using a narrow bandpass filter aligned in 
frequency to L. The raw sampling rate has to be twice the requirement set by the 
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object tracking because two frames are differenced to get one frame to supply the 
face recognition software. The LEDs 30 have to be much faster. The IR radiation 
source has to be modulated such that all emission is during the time when all 
detectors are active. If the integration times of all pixels in the detector are not 
5 aligned, the available time for the source to be ON is reduced by the worst case 
misalignment. 

The face recognition system described above for identifying and tracking a 
person in a vehicle passenger seat can be extended to other types of vehicle 
systems for other purposes. Figure 8 is a top plan view of a vehicle 110 showing 

10 sensors or cameras positioned at various locations for other purposes. For 
example, a forward looking infrared sensor 112 is positioned at a center location 
on the instrument panel 22 and is directed forward for detecting objects in front of 
the vehicle. Additionally, a rearward looking infrared sensor 114 is positioned at a 
center location on the instrument panel 22 'and is directed towards the rear 

15 passenger compartment of the vehicle 110 to monitor activity in that part of the 
vehicle. Additionally, an infrared sensor 116 is positioned on the instrument panel 
22 in front of the driver seat 118 to detect the driver's facial features for various 
purposes discussed below. An infrared sensor 120 is positioned on the 
instrument panel 22 in front of the passenger seat 122 for airbag deployment as 

20 discussed above. The various sensors 112, 114, 116 and 120 can be used in 
different ways in different systems. Other sensor locations can also be employed 
for other purposes consistent with the discussion herein. Some of the applications 
are discussed below. 

In one embodiment, according to the present invention, a sensor or camera 

25 is pointed towards the passenger and driver's seat that takes an image of the 
driver and the passenger's facial feature. Using the facial feature software, the 
contours of the face and eye coordinates of the driver and the passenger are 
compared to images stored in a computer data base or lookup table. If the 
software makes a match of the human face with one stored in the data base, the 

30 driver and/or passenger airbag will be adjusted so that that person's 
characteristics are used to personalize airbag deployment. This will help the 
airbag to automatically adjust for low weight, young persons, short drivers, etc. 
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According to the invention, a drowsy driver application involves identifying 
when the driver is falling asleep at the wheel and waking the driver up. Initially, at 
regular time intervals, the face recognition software acquires the facial features 
such as. eyes, or any combination of 2-D or 3-D facial/head/torso feature 
5 coordinates of the driver. Depending on the new coordinates of the driver's facial 
features, the software will send a warning to the driver that he or she is falling 
asleep. This warning will be based on the frequency of the driver's head nods, 
i.e., the back and forth head motion, whether the driver's eyes are closed, etc. In 
particular, the percentage of time that the eyes are closed during a fixed time 

10 interval is used to determine if the driver is asleep. Preliminary studies have 
shown that when a driver's eyes are shut more than 70% of the time over two 
seconds, the driver is drowsy and may be falling asleep. In addition, the back and 
forth motion of the driver's head or the driver's eyes looking at a cell phone will be 
utilized to determine if the driver is paying attention to the road ahead. The face 

15 recognition software can sense the difference in the driver's eye coordinates and 
this information can be relayed back to the driver. This information returns to the 
driver in the form of a mechanical motion, or an electrical stimulus or the 
production of a loud signal to wake the driver. 

Similar to the drowsy driver application discussed above, drug use could 

20 cause the driver to fall asleep. Presently, many drivers use caffeine or speed to 
keep them awake over a long trip. However, when the drug wears off, the driver's 
response time diminishes to critical changes in road or driving conditions. Thus, 
monitoring of the driver's pupils will help detect a driver using drugs when the 
drugs stop working. Basically, the facial feature recognition software determines 

25 how often the iris of the eye is blocked by the driver's eyelid to determine whether 
the driver is falling asleep. 

The facial feature recognition system described herein can also be used for 
driver identification. When the driver enters the vehicle, a camera pointing 
outward placed near the driver's door will take an image of the driver's facial 

30 features. Using the facial feature recognition software, the contours of the facial 
features such as eyes, or any combination of 2-D or 3-D facial/head/torso feature 
coordinates of the perspective driver are compared to images stored in the 

13 



SDOCtO. <WO OPOR022A2 I > 



WO 02/08022 




CTAJSO 1/2 1687 



software data base. If the software makes a match of the face with one stored in 
the data base, the doors will open and the vehicle can be automatically started. 
Otherwise, the vehicle will remain in the off position. In an alternate embodiment, 
after the driver gains access to the vehicle and is in the driver seat, the camera 
5 can be used to identify the driver for purposes of starting the vehicle. Additionally, 
in a parental control situation, the driver can be identified as a young person, 
where the algorithm sets the control of the vehicle to a maximum speed for that 
driver's use. 



10 using the face recognition software as well. Particularly, the driver's human facial 
features can be used to recall the driver's last seat position, rear and side mirror 
adjustment, steering wheel setting, radio setting, etc. In this example, the 
vehicle's computer can store the driver's last seat position, and rear and side 
mirror adjustment with the driver's human facial features. Subsequently, 

15 whenever the camera acquires the driver's facial features and this image matches 
a stored driver's personal preference, the seat and mirrors can be automatically 
repositioned to the desired locations. 

The use of the specific imaging and processing techniques of the present 
invention allows the presence detection of people of all ages and sizes, whether 

20 conscious or asleep, in the interior of a vehicle with very high reliability. This 
detection is effective at all times, for stationary or moving vehicles, whether the 
engine is running or not, without being perturbed by external stimulus to the 
vehicle activities so that determination of dangerous human interior situations is 
enabled to generate warning alerts locally or to be communicated. This includes 

25 situations of abandoned children, medical emergency, captive occupants, sleeping 
occupants, etc. 

Figure 9 is a side view of a vehicle 130 including a forwarded mounted 
sensor 132 that provides a sensor field-of-view in front of the vehicle 130. 
Reflections from objects in the forward looking direction, such as a pedestrian in a 
30 crosswalk, a slow moving vehicle, etc., will be registered by the processing 
system, and provide a warning to the driver of an object in the vehicle's path. By 
recognizing a pedestrian in a crosswalk or the like, pre-crash avoidance can be 



The driver ID concept can be extended to driver convenience features 
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provided using the face recognition system of the present invention. 

Figure 10 is a top view of a vehicle 140 that includes a center mounted 
camera 142 behind the vehicle 140 for providing an indication of objects in the 
vehicle's path when it is moving in a rearward direction. Additionally, a camera 
5 144 placed in the driver's side mirror 146 can detect objects located in the driver's 
blind spot to provide warnings that the driver may not see. In this example, the 
camera 144 will allow the driver to stay focused on the road ahead, but can still 
safely make a right or left turn or lane change. The camera 144 can be provided 
for detecting pedestrians or riders on bicycles in the vehicle blind spot by the face 

10 recognition system. 

In severe automobile collisions, it is sometimes difficult to determine what 
caused the accident. In one example, before the crash event, the exterior forward 
looking camera can make pre-crash images. These images will be relayed back 
to the driver in the form of an electrical or mechanical warning so that the driver 

15 can make adjustments to avoid a collision. In the event of a vehicle collision, the 
images acquired will be stored so at a later date, the pre-crash sensing conditions 
will be available to make a post crash injury assessment. After the crash, a crash 
notification signal can be automatically transmitted in the form of a radio wave 
signal which will alert the police and fire department of the crash location. 

20 To keep the driver's attention on the road and hands on the steering wheel, 

the driver's facial gestures, such as movement of facial features, can be used to 
perform tasks previously performed manually. For example, a driver's facial 
feature command can be compared to an existing stored facial feature that will 
cause the vehicle to automatically change the automobile controls without the 

25 driver taking his or her eyes off the road. Some examples include control of 
automobile accessories, such as turning on and off the radio, turning on and off 
the vehicle lights, turning on and off the windshield wipers or blowing the horn. 
Figure 11 shows a side view of a vehicle 152 including a sensor 154 of the type 
discussed herein positioned in front of a driver 156 for identifying and tracking the 

30 facial features of the driver 156. The face recognition system of the invention can 
use this information in the manner as discussed herein. 

Further, a camera can be positioned in the vehicle to monitor intrusion. In 

15 
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this example, a camera takes an image anywhere inside of the vehicle passenger 
compartment. If a person is detected in the backseat of the vehicle, the camera 
will take an image of the person's facial features. If the facial features match one 
stored in the computer, then no alarm will go off. Otherwise, an alarm can be 
5 sounded to alert the driver that someone is in the back seat. The camera can 
provide a recording of the intruder for later identification, and provide detection of 
human presence after the vehicle is locked. Figure 12 shows a side view of a 
vehicle 160 depicting this example, where a sensor 162 is mounted at a center 
location in the vehicle passenger compartment, such as in the center dome light 

10 The foregoing discussion discloses and describes merely exemplary 

embodiments of the present invention. One skilled in the art will readily recognize 
from such discussion and from the accompanying drawings and claims, that 
various changes, modifications and variations can be made therein without 
departing from the spirit and scope of the invention as defined in the following 

15 claims. 
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CLAIMS 

What is Claimed is: 

1 . An imaging system for a vehicle, said system comprising: 

an infrared source generating an infrared beam along a particular 
field-of-view in or around the vehicle; 
5 an infrared detector responsive to infrared radiation reflected from 

objects in the field-of-view of the infrared beam, said detector generating image 
signals from the reflected infrared radiation; and 

a processor responsive to the image signals from the detector, 
said processor including face recognition software that acquires and tracks facial 
10 features of a person from the image signals, said processor causing a vehicle 
operation to be performed if a person's face is detected. 

2. The system according to claim 1 wherein the processor compares 
a detected image of a person's face with stored images to identify the person. 

3. The system according to claim 2 wherein the system is a driver 
15 identification system, said processor identifying a driver's face and causing a 

vehicle door to automatically unlock if the driver's face is detected. 

4. The system according to claim 2 wherein the system is a driver 
identification system, said system identifiying a driver's face, said system 
monitoring driver gestures to perform a particular vehicle function if the driver's 

20 face is detected. 

5. The system according to claim 1 wherein the system is a vehicle 
convenience system, said processor identifying a person's face and causing a 
vehicle system to be automatically adjusted in response to detecting the 
person's face. 

25 6. The system according to claim 2 wherein the system records the 

person's face if it does not match a face stored in the processor. 

7. The system according to claim 2 wherein the processor causes the 
vehicle to start if the detected image matches a particular stored image. 

8. The system according to claim 2 wherein the processor controls 
30 the vehicle speed if a detected image is of .a particular person's face that is 

stored. 
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9. The system according to claim 1 wherein the processor tracks and 
monitors the persons eyes. 

10. The system according to claim 1 further comprising an infrared 
filter positioned in front of the detector. 

5 11. The system according to claim 1 wherein the infrared detector 

includes a pixel array of photodiodes. 

12. The system according to claim 1 wherein the infrared source is 
pulsed on for a predetermined time at predetermined intervals. 

13. The system according to claim 12 wherein the pulsed source and 
10 the detector are synchronized such that a difference of successive frames 

represents an image free of ambient light variations. 

14. The system according to claim 1 wherein the detector includes a 
pixel array of diodes. 

15. A driver imaging and identification system for use in a vehicle, said 
15 system comprising: 

an infrared source generating an infrared beam along a particular 
field-of-view in or around the vehicle, wherein the infrared source is pulsed on 
and off at predetermined intervals; 

an infrared detector responsive to infrared radiation reflected from 
20 objects in the field-of-view of the infrared beam, said detector generating image 
signals from the reflected radiation; and 

a processor responsive to the image signals from the detector, 
said processor including face recognition software that acquires and tracks facial 
features of a person from the image signals and compares a detected image of 
25 a person's face with stored images to identify the person, said processor 
identifying a driver's face from the comparison and causing a vehicle door to 
automatically unlock if the driver's face is detected. 

16. An intruder detection system for use in a vehicle, said system 
comprising: 

30 an infrared source generating an infrared beam along a particular 

field-of-view in or around the vehicle, wherein the infrared source is pulsed on 
and off at predetermined intervals; 
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an infrared detector responsive to infrared radiation reflected from 
objects in the field-of-view of the infrared beam, said detector generating image 
signals from the reflected radiation; and 

a processor responsive to the image signals from the detector, said 
5 processor including face recognition software that acquires and tracks facial 
features of a person from the image signals and compares a detected image of a 
person's face with stored images to identify the person, wherein if the identified 
face does not match a stored image, the processor causes an alarm to sound. 

17. The system according to claim 16 wherein the processor provides a 
10 recording of the image if the detected image does not match a stored image. 

18. A driver convenience system for use in a vehicle, said system 
comprising: 

an infrared source generating an infrared beam along a particular 
field-of-view in or around the vehicle, wherein the infrared source is pulsed on 
1 5 and off at predetermined intervals; 

an infrared detector responsive to infrared radiation reflected from 
objects in the field-of-view in or around the vehicle; 

an infrared detector responsive to infrared radiation reflected from 
objects in the field-of-view of the infrared beam, said detector generating image 
20 signals from the reflected radiation; and 

a processor responsive to the image signals from the detector, said 
processor including face recognition software that acquires and tracks facial 
features of a person from the image signals and compares a detected image of a 
person's face with stored images to identify the person, said processor causing a 
25 vehicle system to be automatically set in response to detecting the person's face. 

19. A driver imaging and identification system for use in a vehicle, said 
system comprising: 

an infrared source generating an infrared beam along a particular 
field-of-view in or around the vehicle, wherein the infrared source is pulsed on 
30 and off at predetermined intervals; 

an infrared detector responsive to infrared radiation reflected from 
objects in the field-of-view of the infrared beam, said detector generating image 
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. signals from the reflected radiation; and 

a processor responsive to the image signals from the detector, 
said processor including face recognition software that acquires and tracks facial 
features of a person from the image signals and compares a detected image of 
5 a person's face with stored images to identify the person, said processor 
identifying a driver's face from the comparison and monitoring driver commands 
to perform a particular vehicle function. 

20. The system according to claim 19 wherein the processor detects the 
person after the vehicle is locked. 
10 21. A driver convenience system for use in a vehicle, said system 

comprising: 

an infrared source generating an infrared beam along a particular 
field-of-view in or around the vehicle, wherein the infrared source is pulsed on 
and off at predetermined intervals; 

15 an infrared detector responsive to infrared radiation reflected from 

objects in the field-of-view in or around the vehicle; 

an infrared detector responsive to infrared radiation reflected from 
objects in the field-of-view of the infrared beam, said detector generating image 
signals from the reflected radiation; and 

20 a processor responsive to the image signals from thie detector, said 

processor including face recognition software that acquires and tracks facial 
features of a person from the image signals and compares a detected image of a 
person's face with stored images to identify the person, said processor causing 
the vehicle to start if the detected image is determined to be the vehicle driver. 

25 22. A driver convenience system for use in a vehicle, said system 

comprising: 

an infrared source generating an infrared beam along a particular 
field-of-view in or around the vehicle, wherein the infrared source is pulsed on 
and off at predetermined intervals; 
30 an infrared detector responsive to infrared radiation reflected from 

objects in the field-of-view in or around the vehicle; 

20 
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an infrared detector responsive to infrared radiation reflected from 
objects in the field-of-view of the infrared beam, said detector generating image 
signals from the reflected radiation; and 

a processor responsive to the image signals from the detector, said 
5 processor including face recognition software that acquires and tracks facial 
features of a person from the image signals and compares a detected image of a 
person's face with stored images to identify the person, said processor causing 
the vehicle speed to be controlled in response to detecting a particular person. 
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